Use official TRT-LLM image (1.3.0rc15.post1) for DSv4 B300 TRT (non-MTP + MTP)#1636
Use official TRT-LLM image (1.3.0rc15.post1) for DSv4 B300 TRT (non-MTP + MTP)#1636Oseltamivir wants to merge 18 commits into
Conversation
…03e6 Bumps the TensorRT-LLM DeepSeek-V4-Pro image for dsv4-fp4-b200-trt and dsv4-fp4-b300-trt to ghcr.io#semianalysisai/trtllm-deepseek-v4:feat-deepseek_v4-2dd03e6. The -mtp variants are intentionally left on feat-deepseek_v4-9aa3715. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
Thanks for the contribution! For vLLM & SGLang, please ensure that your recipes is similar to the official vLLM recipes and/or the SGLang cookbook If it is not, please create a PR first before we can merge your single node PR into the master branch. Let's ensure that the documentation is first class such that the entire ML community can benefit from your hard work! Thank you PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. If re-running failed jobs is attempted, PR authors are responsible for ensuring it passes. See GitHub's docs on re-running failed jobs: https://docs.github.com/en/actions/how-tos/manage-workflow-runs/re-run-workflows-and-jobs#re-running-failed-jobs-in-a-workflow As a rule of thumb, generally, PR authors should request a review & get a PR approval from the respective companies' CODEOWNERS before requesting a review from core maintainers. If additional help is needed, PR authors can reach out to core maintainers over Slack. |
1 similar comment
|
Thanks for the contribution! For vLLM & SGLang, please ensure that your recipes is similar to the official vLLM recipes and/or the SGLang cookbook If it is not, please create a PR first before we can merge your single node PR into the master branch. Let's ensure that the documentation is first class such that the entire ML community can benefit from your hard work! Thank you PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. If re-running failed jobs is attempted, PR authors are responsible for ensuring it passes. See GitHub's docs on re-running failed jobs: https://docs.github.com/en/actions/how-tos/manage-workflow-runs/re-run-workflows-and-jobs#re-running-failed-jobs-in-a-workflow As a rule of thumb, generally, PR authors should request a review & get a PR approval from the respective companies' CODEOWNERS before requesting a review from core maintainers. If additional help is needed, PR authors can reach out to core maintainers over Slack. |
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=26783090679 |
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
|
|
||
| dsv4-fp4-b200-trt: | ||
| image: ghcr.io#semianalysisai/trtllm-deepseek-v4:feat-deepseek_v4-9aa3715 | ||
| image: ghcr.io#semianalysisai/trtllm-deepseek-v4:feat-deepseek_v4-2dd03e6 |
There was a problem hiding this comment.
is there any official nvidia RC that works...
There was a problem hiding this comment.
Image is from dsv4 branch: https://github.com/NVIDIA/TensorRT-LLM/tree/feat/deepseek_v4
Main dsv4 failing DPA: https://github.com/SemiAnalysisAI/InferenceX/actions/runs/26786937394
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=26783097365 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=26786056973 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=26786107993 |
… (non-MTP) Swap dsv4-fp4-b200-trt and dsv4-fp4-b300-trt from the custom ghcr.io semianalysis feat/deepseek_v4 build to the official nvcr.io#nvidia/tensorrt-llm/release:1.3.0rc15.post1 to test whether the official RC can serve DeepSeek-V4-Pro. The -mtp variants are unchanged. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Oseltamivir
changed the title
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=26786937394 |
…on-MTP) The official nvcr.io tensorrt-llm/release:1.3.0rc15.post1 loads DSv4-Pro but its DP-attention path deadlocks/crashes under concurrent load (every dpa=true job hung or failed; only pure-TP conc-1 points passed). Revert to the stable custom build until upstream fixes DSv4 + attention-DP (NVIDIA/TensorRT-LLM#13431). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Oseltamivir
changed the title
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=26803566770 |
1 similar comment
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=26803566770 |
Bump dsv4-fp4-b200-trt and dsv4-fp4-b300-trt to ghcr.io#semianalysisai/trtllm-deepseek-v4:fix-dsv4-swa-scratch-revert-shrink-c914d6d (TRT-LLM feat/deepseek_v4 @ 084cf2ba + kv_cache_manager_v2 fix). This resolves the engine crash on attention-DP context/generation reverts at high concurrency (the b300 8k1k conc>=512 "LLM is shutting down" hang). The -mtp variants stay on feat-deepseek_v4-9aa3715. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Oseltamivir
changed the title
B200 reverts to feat-deepseek_v4-9aa3715: the 2dd03e6 image OOMs on B200's smaller HBM at conc-256 once SWA scratch reuse is disabled. Only B300 moves to 2dd03e6 + TRTLLM_DSV4_ENABLE_SWA_SCRATCH_REUSE=0 in its launcher. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Oseltamivir
changed the title
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=26912996470 |
…TP + MTP) Point dsv4-fp4-b300-trt and dsv4-fp4-b300-trt-mtp at the official nvcr.io#nvidia/tensorrt-llm/release:1.3.0rc15.post1 (from the custom feat/deepseek_v4 builds 2dd03e6 / 9aa3715) and drop the TRTLLM_DSV4_ENABLE_SWA_SCRATCH_REUSE=0 launcher workaround so the official image runs with native behavior. B200 TRT unchanged (9aa3715). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Oseltamivir
changed the title
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=26914210927 |
1 similar comment
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=26914210927 |
…PA crash The previous sweep crashed all dpa=true jobs with CUDA_ERROR_ILLEGAL_ADDRESS on rc15.post1 without the SWA scratch workaround. Re-add TRTLLM_DSV4_ENABLE_SWA_SCRATCH_REUSE=0 to both B300 TRT launchers (non-MTP and MTP) to determine whether the DPA crash is the same SWA-scratch bug or a separate FMHA kernel issue. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=26999118817 |
…A crash) TRTLLM_DSV4_ENABLE_SWA_SCRATCH_REUSE=0 did not fix the DPA crash on rc15.post1 — the CUDA_ERROR_ILLEGAL_ADDRESS persists during warmup. This is a separate bug from the SWA scratch revert issue. Default back to 1 (native behavior). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
| - config-keys: | ||
| - dsv4-fp4-b300-trt | ||
| - dsv4-fp4-b300-trt-mtp | ||
| description: | ||
| - "Point the B300 TensorRT-LLM DeepSeek-V4-Pro configs (non-MTP dsv4-fp4-b300-trt and MTP dsv4-fp4-b300-trt-mtp) at the official NVIDIA release image nvcr.io/nvidia/tensorrt-llm/release:1.3.0rc15.post1, replacing the custom ghcr.io semianalysis feat/deepseek_v4 builds (2dd03e6 and 9aa3715 respectively), to evaluate the official RC for DeepSeek-V4-Pro. Also drops the TRTLLM_DSV4_ENABLE_SWA_SCRATCH_REUSE=0 launcher workaround (specific to the custom build) so the official image runs with its native behavior. B200 TRT is unchanged (stays on feat-deepseek_v4-9aa3715)." | ||
| pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/1636 |
There was a problem hiding this comment.
🟡 Duplicate and contradictory perf-changelog entries for PR #1636: two entries cover the same config-keys (dsv4-fp4-b300-trt and dsv4-fp4-b300-trt-mtp) with contradictory descriptions. The first entry (lines 3390-3395) claims the PR "drops the TRTLLM_DSV4_ENABLE_SWA_SCRATCH_REUSE=0 launcher workaround ... so the official image runs with its native behavior", but the same PR actually ADDS that export with default 0 to both launchers (dsv4_fp4_b300_trt.sh L62-65 and dsv4_fp4_b300_trt_mtp.sh L61-64). The second entry (lines 3455-3460) is correct; please remove the first (stale) entry so the changelog has a single, accurate source of truth.
Extended reasoning...
What the bug is
perf-changelog.yaml ends up with two distinct entries for the same PR (#1636) and the same config-keys (dsv4-fp4-b300-trt, dsv4-fp4-b300-trt-mtp), and the two entries say opposite things about whether the TRTLLM_DSV4_ENABLE_SWA_SCRATCH_REUSE=0 launcher workaround is shipped.
- Entry A (perf-changelog.yaml:3390-3395): "Point the B300 TensorRT-LLM DeepSeek-V4-Pro configs ... at the official NVIDIA release image ... Also drops the TRTLLM_DSV4_ENABLE_SWA_SCRATCH_REUSE=0 launcher workaround (specific to the custom build) so the official image runs with its native behavior. B200 TRT is unchanged."
- Entry B (perf-changelog.yaml:3455-3460): "Switch B300 DSv4 TRT (non-MTP + MTP) to official rc15.post1 image with TRTLLM_DSV4_ENABLE_SWA_SCRATCH_REUSE=0 to test whether DPA crash is same SWA-scratch bug".
Why entry A is wrong
The same PR diff adds the export to both launcher scripts with default 0:
# benchmarks/single_node/fixed_seq_len/dsv4_fp4_b300_trt.sh:62-65
export TRTLLM_DSV4_ENABLE_SWA_SCRATCH_REUSE="${TRTLLM_DSV4_ENABLE_SWA_SCRATCH_REUSE:-0}"
echo "TRTLLM_DSV4_ENABLE_SWA_SCRATCH_REUSE: $TRTLLM_DSV4_ENABLE_SWA_SCRATCH_REUSE"
# benchmarks/single_node/fixed_seq_len/dsv4_fp4_b300_trt_mtp.sh:61-64 (identical)Because the parameter expansion ${VAR:-0} defaults to 0 when unset, sweeps using the official rc15.post1 image still disable SWA scratch reuse — the workaround is kept, not dropped. The recent commit c2381b7 ("Re-add TRTLLM_DSV4_ENABLE_SWA_SCRATCH_REUSE=0 on rc15.post1 to test DPA crash") confirms this is the intended shipped behavior, matching Entry B and contradicting Entry A.
Impact
Doc-only — the runtime behavior is unambiguously the one described in Entry B (the workaround is on). But the changelog is the project's audit trail: any human or tool that consumes it to ask "what did PR #1636 ship for dsv4-fp4-b300-trt?" now gets two contradictory answers for the same config-keys. Future debugging of the rc15.post1 DPA crash will be harder because the changelog suggests the workaround was removed when it actually was retained — exactly the opposite of the diagnostic intent.
Fix
Delete Entry A (perf-changelog.yaml:3389-3395). Entry B already covers the same configs with the accurate description. Independently flagged by Cursor Bugbot in two comments on this PR: Duplicate PR changelog entries (Low) and Changelog contradicts launcher workaround (Medium).
Step-by-step proof
- Open
perf-changelog.yaml, scroll to line 3389. Entry A listspr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/1636withconfig-keys: [dsv4-fp4-b300-trt, dsv4-fp4-b300-trt-mtp]. - Scroll to line 3454. Entry B lists the same
pr-linkand the sameconfig-keys. - Read Entry A's description — it says the SWA workaround is dropped.
- Read Entry B's description — it says the SWA workaround is set to
0to test DPA crash. - Open
benchmarks/single_node/fixed_seq_len/dsv4_fp4_b300_trt.shat line 62. Observeexport TRTLLM_DSV4_ENABLE_SWA_SCRATCH_REUSE="${TRTLLM_DSV4_ENABLE_SWA_SCRATCH_REUSE:-0}"— defaults to0. - Same in
dsv4_fp4_b300_trt_mtp.shat line 61. - Therefore the shipped behavior matches Entry B; Entry A is stale and should be removed.
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
There are 2 total unresolved issues (including 1 from previous review).
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 65bddbb. Configure here.
|
|
||
| # Disable DSv4 SWA scratch reuse to test whether the rc15.post1 DPA crash | ||
| # is the same SWA-scratch bug or a separate FMHA kernel issue. | ||
| export TRTLLM_DSV4_ENABLE_SWA_SCRATCH_REUSE="${TRTLLM_DSV4_ENABLE_SWA_SCRATCH_REUSE:-1}" |
There was a problem hiding this comment.
SWA default contradicts disable comment
Medium Severity
The new launcher comments say DSv4 SWA scratch reuse is disabled to isolate the rc15.post1 DPA crash, but TRTLLM_DSV4_ENABLE_SWA_SCRATCH_REUSE defaults to 1 when unset, which keeps scratch reuse enabled. Sweep runs without an override therefore do not match the stated experiment.
Additional Locations (1)
Reviewed by Cursor Bugbot for commit 65bddbb. Configure here.
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=27034741001 |
|
@claude try using nvcr.io#nvidia/ai-dynamo/tensorrtllm-runtime:1.3.0-deepseek-v4-dev.1 |
|
Claude finished @functionstackx's task in 2m 2s —— View job Switched DSv4 B300 TRT image to
|
…ntime:1.3.0-deepseek-v4-dev.1 Update both dsv4-fp4-b300-trt and dsv4-fp4-b300-trt-mtp to the Dynamo TensorRT-LLM runtime image. Also removes stale perf-changelog entry that contradicted the launcher behavior. Co-authored-by: functionstackx <functionstackx@users.noreply.github.com>
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=27174218420 |
|
Something still broken at high conc |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=27174218420 |
3 participants
Points both B300 DSv4 TRT configs at the official NVIDIA release image and adds the MTP sibling to the sweep:
dsv4-fp4-b300-trt(non-MTP):feat-deepseek_v4-2dd03e6→nvcr.io#nvidia/tensorrt-llm/release:1.3.0rc15.post1dsv4-fp4-b300-trt-mtp(MTP):feat-deepseek_v4-9aa3715→nvcr.io#nvidia/tensorrt-llm/release:1.3.0rc15.post1This drops the custom
ghcr.iosemianalysisfeat/deepseek_v4builds in favor of the official RC, to evaluate whether the official image can serve DeepSeek-V4-Pro (non-MTP and MTP). The non-MTP launcher'sTRTLLM_DSV4_ENABLE_SWA_SCRATCH_REUSE=0workaround (specific to the custom build) is removed so the official image runs with its native behavior, matching the MTP launcher which never had it.Known risk
A prior run of
1.3.0rc15.post1with attention-DP (dpa=true) served a couple of iterations and then crashed withCUDA_ERROR_ILLEGAL_ADDRESSinkv_cache_manager.free_resources(run 26786937394) — a different failure from the custom build's SWA-scratch-revert crash. Sodpa=truejobs may still fail on the official image; the pure-TP (dpa=false) cases are more likely to pass. MTP on the official RC is untested. This sweep is what tells us where it stands.Scope
B200 TRT is unchanged (stays on
feat-deepseek_v4-9aa3715); its OOM follow-up is tracked separately.🤖 Generated with Claude Code
Note
Medium Risk
Benchmark-only change, but it swaps production-like runtime images and exercises unstable DSv4 TRT paths (DPA/MTP) where prior official builds hit CUDA illegal-address failures.
Overview
Moves B300 DeepSeek-V4 FP4 TensorRT-LLM benchmarks (
dsv4-fp4-b300-trtanddsv4-fp4-b300-trt-mtp) off customghcr.io/semianalysisaiimages ontonvcr.io/nvidia/ai-dynamo/tensorrtllm-runtime:1.3.0-deepseek-v4-dev.1innvidia-master.yaml.Both fixed-seq-len launchers now export
TRTLLM_DSV4_ENABLE_SWA_SCRATCH_REUSE(default 1, overridable via env) and log it, so sweeps can toggle SWA scratch reuse and see whether attention-DP crashes match the old SWA-scratch failure or a separate FMHA issue.perf-changelog.yamlrecords the image switch and the scratch-reuse experiment.Reviewed by Cursor Bugbot for commit 485aacb. Bugbot is set up for automated code reviews on this repo. Configure here.